R-TF-028-004 Data Annotation Instructions - Non-clinical data
Table of contents
Context
This document provides the instructions on how to annotate dermatological images for non-clinical data. The annotation process is distinct from the ICD-11, Binary Indicator, and Visual Sign labeling, and involves several different tasks, including visual image quality scoring, image type, skin type and body site identification, and drawing bounding boxes for specific body parts. This process is fundamental for training the supervised learning models that contribute to the overall functioning of the device.
Objectives
- To create a comprehensive, multi-modal ground-truth dataset for the quantification and description of non-clinical attributes of the images.
- To provide clear, unambiguous instructions to medical annotators to ensure consistency and accuracy across all annotation tasks.
- To establish a robust foundation for training, validating, and testing the AI/ML models responsible for non-clinical data analysis.
Annotation Personnel
Role
Data annotation professional.
Qualifications
- Required: Normal visual aquity and average computer skills.
- Experience: A minimum of three years of experience in data annotation services.
Responsibilities
- To review and understand these annotation instructions thoroughly.
- To perform annotations on the provided images using the designated annotation platform.
- To complete each annotation task accurately and consistently according to the specific guidelines below.
General Annotation Workflow
All annotations will be performed using a web-based medical image annotation platform. For each image assigned, the annotator will:
- Examine the image to identify all relevant features related to the task at hand.
- Perform the annotation as per the specific instructions for that task type (visual quality assessment, image domain, skin type and body site classification, and body part bounding box annotation).
- Repeat the process until all non-clinical labels have been annotated.
Specific Annotation Instructions by Task Type
This section details the distinct annotation methodologies for each category of visual sign.
Task: Image Quality Assessment (Ordinal)
- Instruction: Select the visual quality category from the provided list that best represents the perceived visual quality of the image.
| Rating | Visual quality |
|---|---|
| 1 | Bad |
| 2 | Poor |
| 3 | Fair |
| 4 | Good |
| 5 | Excellent |
Task: Image domain (Categorical)
- Instruction: Label the image as one of the following categories:
- Clinical image
- Dermoscopy image
- Histology image
- Skin image but non-dermatology
- Out-of-domain image
Task: Skin type (Ordinal)
This task is done using both the Fitzpatrick skin type (FST) and Monk skin type (MST) scales.
- Instruction (FST): Observe the skin content of the image and label the skin type as one of the following: I, II, III, IV, V, VI.
- Instruction (Monk): Observe the skin content of the image and label the skin type as one of the categories in the Monk scale (1 to 10)
| Skin tone group | FST | MST |
|---|---|---|
| Light | I-II | 1-3 |
| Medium | III-IV | 4-6 |
| Dark | V-VI | 7-10 |
Task: Image domain (Categorical)
- Instruction: Inspect the image and identify which body parts are visible, and then label the image as one or more of the following categories:
- Scalp
- Top of the head
- Back of the head
- Face
- Mouth
- Tongue
- Ear
- Eye
- Nose
- Neck
- Trunk, chest and abdomen
- Back
- Armpit
- Arm
- Hand
- Finger
- Hand nail
- Leg
- Knee
- Foot
- Toe
- Foot nail
- Genitals (groin, penis, vulva, anus)
- Buttock
- Close-up image (body parts not visible)
Task: Head localization (Bounding Boxes)
- Instruction: I a head is visible in the image, use the bounding box tool to carefully draw a bounding box that encloses the person's head.
- Guidelines:
- The outline should be as precise as possible, tightly enclosing the skull.
- Place polygon points close enough together to accurately capture curves.
- For conditions with multiple heads (e.g. the image is a mosaic of different head views), draw a separate bounding box for each head.
Quality Control
To ensure the creation of a high-fidelity ground truth and to measure inter-observer variability, a rigorous quality control process will be followed.
- Multi-Annotator Process: To ensure the highest quality, images will be annotated by one or more qualified annotators. In the case of image quality assessment (IQA), at least 15 observers will be used for each annotation project, following the ITU-T P.910 recommendation from the International Telecommunications Union (ITU).
- Consensus Ground Truth: When multiple experts annotate the same image, the final ground-truth label will be established by pooling their independent assessments. The method for establishing consensus will depend on the task (e.g., mean for image quality scores, voting for categories, or algorithmic fusion for bounding boxes/polygons).
Signature meaning
The signatures for the approval process of this document can be found in the verified commits at the repository for the QMS. As a reference, the team members who are expected to participate in this document and their roles in the approval process, as defined in Annex I Responsibility Matrix of the GP-001, are:
- Author: Team members involved
- Reviewer: JD-003, JD-004
- Approver: JD-001